Systems and methods for capturing critical fields from a mobile image of a credit card bill

ABSTRACT

The embodiments herein focus on improving recognition accuracy of these fields on credit card bills by detecting and identifying critical fields on a credit card, extracting the data from the critical fields and comparing the data with known data on a payor, payee and biller.

BACKGROUND

1. Field of the Invention

The embodiments described herein relate to processing images capturedusing a mobile device, and more particularly to identifying criticalfields in a credit card remittance coupon and extracting the contenttherein.

2. Related Art

Financial institutions which issue credit cards frequently offer aservice known as a balance transfer, where a customer with a balance dueon a credit card can transfer some or all of the outstanding balancefrom one credit card to another credit card. Customers typicallytransfer balances from one card to another to obtain a lower interestrate, more favorable payment schedule, or other benefits offered by acredit card for carrying a balance with a particular financialinstitution. A balance transfer may also be similar to a cash advance,where a customer can transfer a sum of money from their credit card intotheir bank account, resulting in a balance due on the credit card butgiving the customer cash in their bank account.

In some situations, the customer already holds the credit card where thebalance is being transferred, while in other situations, the customermay be opening a new credit card and transferring a balance to the newcredit card. Banks often compete with other banks to advertise lowerinterest rates and favorable payment terms on a balance transfer.However, it is often difficult for a customer to find out which balancetransfer offers are available and what the terms of the balance transferwill be, as many balance transfer terms are dependent on the amount ofthe balance being transferred or the credit rating of the customer.

The balance transfer process is cumbersome for both the customer and thebank. The customer must obtain several different pieces of information,including the customer's name, contact information, credit card number,the current balance and the applicable interest rates that areapplicable to the balance. If the balance is being transferred to a bankaccount, other information may be needed, such as a bank account numberand routing number. A bank may also want to evaluate the credit historyof the customer to determine whether to accept the balance transferapplication, in which case the customer will need to provide even moreinformation, such as a social security number, driver's license numberor additional financial information.

Once this information is entered into an application for a balancetransfer, the receiving bank evaluates the information to determinewhether to accept the balance transfer request. This process may take asignificant amount of time—generally several days. Once accepted, it maytake several more day or even weeks before the money is transferred.

Therefore, there is a need for streamlining the process of applying forand processing financial offers, such as credit card balance transfers.

SUMMARY

Embodiments described herein provide for the identification of criticalfields on a document which provide high probabilities of accuratelyreading a content on the document image. By improving recognitionaccuracy of these fields on documents such as a credit card bill, theremainder of the content on the bill can be read with high confidence.

Products which use image processing techniques to read bills, includingsuch bill categories as insurance, utility, mortgage etc., use a set ofrules which apply to all (or majority) of bills within each category.One of the most important tasks behind the mobile image capture scienceis understanding and utilization of the category-specific rules in formof specialized OCR, cross-validation between different document fields,usage of postal barcodes etc. For example, knowledge that the documentis a credit card bill (CCB) allows the system to read its Account Numberand other critical fields using both data on the bill and the code-lineand in some cases the code-line only. This reduces the error rate oncritical fields by 2-5 times compared to “generic” bills.

The following fields on CCBs are considered critical on CCBs: AccountNumber, Balance Due, Payee ZIP-code and Biller's Name.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments disclosed herein are described in detail withreference to the following figures. The drawings are provided forpurposes of illustration only and merely depict typical or exemplaryembodiments. These drawings are provided to facilitate the reader'sunderstanding and shall not be considered limiting of the breadth,scope, or applicability of the embodiments. It should be noted that forclarity and ease of illustration these drawings are not necessarily madeto scale.

FIG. 1 is an image of a credit card bill identifying one or morecritical fields, according to embodiments.

FIG. 2 is an image of a portion of the credit card bill used forcross-validation of an account number field against a Codeline,according to embodiments.

FIG. 3 is an image of the credit card bill identifying a payee blockwhich is not sufficiently isolated from other text, causing text blocksegmentation issues, according to embodiments.

FIG. 4 is an illustration of a processed image of a credit card billidentifying a last text line in one or more address boxes, according toembodiments.

FIG. 5 is an image of the credit card bill highlighting a paymenthistory deduction tool to improve accuracy of capturing critical fields,according to embodiments.

FIG. 6 is an image of a portion of the credit card bill highlighting abalance due critical field, according to embodiments.

FIG. 7 is an illustration of a method of identifying a biller's namefield on different portions of the credit card bill, according toembodiments.

FIG. 8 illustrates a method of identifying one or more fields on thecredit card statement, according to one embodiment of the invention.

FIG. 9 illustrates a system for capturing and identifying criticalfields on a credit card statement, according to one embodiment of theinvention.

FIG. 10 is a block diagram that illustrates an embodiment of acomputer/server system upon which an embodiment of the inventivemethodology may be implemented.

The various embodiments mentioned above are described in further detailwith reference to the aforementioned figured and the following detaileddescription of exemplary embodiments.

DETAILED DESCRIPTION

Embodiments described herein pertain to systems and methods foridentifying and capturing critical fields on an image of a document suchas a credit card bill. Each critical field is identified as such basedon the resulting likelihood that if the critical field can beidentified, the remaining fields on the document can also be identifiedwith a high confidence. This therefore improves the overall ability tocapture and identify content from the document and utilize it forvarious applications.

The following fields on CCBs are considered critical for BT application:

Account Number

Balance Due

Payee ZIP-code

Biller's Name

The embodiments herein focus on improving recognition accuracy of thesefields on credit card bills. The following document fields are beingcaptured and used to facilitate finding, identification, and recognitionof critical fields:

105 Codeline

106 Payor Block

107 Postal Barcodes

I. Capturing Account Number from Credit Cards Bills

Keyword-Based Search

AccountNumber field has a unique set of keyword phrases which allow toidentify the field's location on about 90% of CCBs. In remaining 10%,the keyword cannot be found due to some combination of poor imagequality, small font, inverted text etc. Below we discuss methods whichcan be used when keywords cannot be found.

The most frequent keyword phrases are: Account, Account Number andAccount No. Some keyword phrases could be printed in a single text lineor in two consecutive lines. It should be noted that the set of keywordson CCBs is more restrictive than in the general case. For example, suchphrase as “Policy Number” (frequent on insurance bills) is not used.

Keywords are searched for in the full-page OCR result using FuzzyMatching technique. For example, if OCR result contains “AccountNomber”, then the “Account Number” keyword will be found with confidenceof approximately 920 (out of 1000 max) because 12 out of 13 non-spacecharacters are the same as in the “Account Number”. On FIG. 2, thekeyword 201 “Account Number” was found.

Format-Based Search

The data format of Account Number field on CCBs is more restrictive thanin the “generic” bill case both in terms of its length and characterset. More limitations apply in case of Major Credit Cards, see section6.

The data format of Account Number field on CCBs is more restrictive thanin the “generic” bill case both in terms of its length and characterset.

For example, the following definition of Account Number format coversmajority of ALL bills:

Total number of characters: from 4 to 22 Number of low-case alphacharacters (excluding ‘x’): 0 Number of upper-case alpha characters(excluding ‘X’): from 0 to 4 Number of punctuations (spaces, dashes):from 0 to 4 Number of masking characters (X, x, *, #): from 0 to 12

In contrast, the following (narrower) definition of Account Numberformat covers majority of credit card bills:

Total number of characters: from 10 to 20 Number of low-case alphacharacters (excluding ‘x’): 0 Number of upper-case alpha characters(excluding ‘X’): 0 Number of punctuations (spaces, dashes): from 0 to 4Number of masking characters (X, x, *, #): from 0 to 12

The data could be found in proximity to keywords found in 1.1 ordirectly in the full-page OCR result.

Each location of data is assigned the format-based confidence, whichreflects how close data in the found location matches the expectedformat. On FIG. 2, the data 202 was found.

Cross Validation Against Codeline

On CCBs, Account Number is always included into the Codeline. Thisallows to use cross-validation technique.

On CCBs, AccountNumber is always included into the Codeline. This allowsto use cross-validation technique, which works as follows:

Account Number is captured using keywords and/or data formatsdefinition, see 1.1-1.2. Let us refer to an Account Number result as A,see 202 on FIG. 2. Codeline is captured using an OCRA/OCRB recognitionmodule. Let us refer to codeline string as B, see 202 on 203 on FIG. 2.

Substrings of B are compared to A after removing spaces, dashes andother non-essential punctuation marks in both A and B. The matching isdone using Fuzzy Matching technique, explained in [1]. The matchingthreshold is configured in such a way that a single-character differencebetween A and substring of B is allowed. Additional differencesinvolving characters which are frequently misrecognized are alsoallowed. For example, the difference between ‘3’ recognized in aparticular place of A and ‘8’ recognized in the corresponding placeinside B is excused because ‘8’ is frequently recognized as ‘3’. Anotherexample of frequently misrecognized characters are ‘6’ and ‘5’.

If Step (c) finds a substring C which fuzzy-matches A within thethreshold explained above, then A is replaced by the C. The explanationof preferring codeline recognition results to Account Number capturedfrom the bill is that former is significantly more accurate than thelatter.

Consider FIG. 2 as an example. Step 1.1 found the keyword 201, step 2.2found the data 202 (A) next to the keyword. The codeline 203 (B) wasalso found. Let us assume that A is “4888 5755 5555 5561” and B is“43885755555555510000250000126565000000000000006”. The Fuzzy Matchingtechnique (c) will detect that the substring 204 C=“4388575555555551”has 2 differences against A, involving pairs (‘8’ and ‘3’) and (‘6’ and‘5’). Since both pairs are frequently misrecognized (see above) thestring C will be accepted by step (c). As a result, step (d) willcorrect A to “4388 5755 5555 5551.”

AccountNumber's Confidence Score

Each result found by 1.1-1.3 is assigned a confidence score whichreflects how confident the system is that it found a correct fieldresult. In computing the confidence score, a weighted linear combinationof the following factors is used:

-   -   the keyword finding confidence, see 1.1    -   the format-based confidence, see 2.2    -   the score reflecting geometrical alignment between keyword found        in 1.1 and data found by 1.2. For example, if the data is        located immediately to the right from the keyword (with no        characters in between) like 201 and 202 on FIG. 2, or the data        is located immediately below the keyword (again, with no        characters in between), the score reaches its maximum value        of 1000. Various deviation from such alignment or presence of        characters in between cause lower scores.    -   if field's data could be found in the codeline (see 1.3), the        confidence gets an additional boost depending on how well the        data and codeline match.

The weight of each individual factor is the overall field confidencescore is established experimentally.

Cross Validation Against Biller's Database

Available Biller's databases contain information about thousands ofbillers, including their postal addresses, names and account numberformats. This information allows to cross-validate AccountNumber andother critical fields on CCBs, as described in [1]. In case thehighest-confidence AccountNumber result matches Payee informationincluded into the Biller's db, the system will accept the result.However, if it doesn't the system may reconfigure the AccountNumberformat (see 1.2) and try to find it again by repeating steps 1.1 and 1.2or just 1.2 when the new format is significantly more restrictive thanthe default one.

Usage of Specialized OCR

Since the format of Account Number field is more restrictive than in the“generic” bill case, it allows to make OCR more specialized and thus toachieve higher recognition accuracy. For example, a typical OCR error ofmisrecognition of ‘2’ and ‘Z’, ‘O’ and ‘0’, ‘1’ and ‘I’, ‘5’ and ‘S’could be easily avoided if we know that the character is alpha ornumeric.

Usage of Multiple OCR Engines

The system can use multiple OCR engines to recognize and re-recognizesome characters. A typical obstacle to using multiple OCR engines is adifficulty in deciding which one produced correct result. For the samereason as 1.3, making such decision becomes significantly simpler onCCBs.

Using “Last Digits” Hints

If a user enters 1 or more of last digits in the account number, thesystem can utilize such knowledge to improve data capturing accuracy.The mechanism of using such hint is similar to imposing limitations onthe field format (see 1.1). If a user enters one or more of last digitsin the AccountNumber, the system can utilize such knowledge to improvedata capturing accuracy. The mechanism of using such hint is similar toimposing limitations on the field format (see 1.2)

Identification of Major Credit Cards

Since there are very few major credit cards among all credit cardbillers, it is possible to identify the exact major credit usingrelatively simple and fast form identification methods. Such methods arebased on finding logos, certain keyword and overall location of textblocks on the document. Handling of mobile images for the purpose ofForm Identification is described in [2].

Once the system identified that a bill is one of the major creditcards', it can use several rules that apply to such bills (but do notapply to CCBs in general), see Section 6.

Cross Validation Against Biller's Database

Available Biller's databases contain information about thousands ofbillers, including their postal addresses, names and account numberformats. This information allows to cross-validate account number andother critical fields on CCBs.

II. Section 2 Capturing Payee ZIP-Code from Credit Cards Bills

In order to find the Payee ZIP-code and also to ensure its correctness,the system first finds all address blocks on the bill, corrects thoseusing postal barcodes, then identifies which one is Payee and takes itsZIP-code field as the result.

2.1 Using Text Blocks to Find Possible Addresses

Usually, addresses are printed as left-, right- or center-justified textblocks isolated from the rest of document text by significant whitemargins. Based on this information, the system may detect potentialaddress locations on a document by building text block structure. Oneway of doing that is to apply text segmentation features available inmost of OCR systems, such as Fine Reader Engine by ABBYY.

2.2 Using Postal Barcode to Isolate Address Text Blocks

On some bill layouts, where the address blocks are not sufficientlyisolated, the text segmentation method 2.1 may need a correction usingpostal barcodes, as explained on FIG. 4. The bill shown on FIG. 4 hasPayee block 401 printed too close to an unrelated text block 402, whichmay cause a failure of text segmentation algorithm, resulting in wrongtext block 403 being found. To correct the problem, the system can uselocation of postal barcode 404 to define a search area above (shown as405) and below the barcode, thus isolating the correct Payee addressblock 401 from the adjacent text block 402.

2.3 Filtering-Out the Text Blocks by the City/State/ZIP Line

In most of US addresses, the bottommost line contains City/State/ZIPinformation. The system can utilize this knowledge by filtering out thetext blocks found in 2.1-2.2 which do not have enough alphas (torepresent City and State), do not contain any valid state (which isusually abbreviated to 2 characters) and/or do not contain enoughnumbers in the end to represent Zip-code.

Consider the bill shown on FIG. 5: the last text line in two trueaddress text blocks 502 (Payor) and 503 (Payee) contain informationwhich satisfies the conditions described above. Even if OCR makesseveral recognition errors, the Fuzzy Matching algorithm will establisha high degree compliance with expected format. On the other hand, thelast line of text block 501 does not meet the expected format (forexample, none of the last characters in its last row is numeric).Therefore, the text block 501 will be removed from further considerationwhereas blocks 502 and 503 will be both recognized and classified asPayee/Payor as explained in 2.7

2.4. Using Postal Database and Fuzzy Matching to Interpret Addresses

Once address candidates are selected using 2.1-2.3, the BillPay systemcan build the entire address block starting with City/State/ZIP at thebottom line and including 1-3 upper lines as potential Name and StreetAddress components. Since the exact format of the address is notwell-defined (it may have 1-4 lines, be with or without Recipient name,be with or without POBOX etc.), the system has to make multiple addressinterpretation attempts to achieve satisfactory interpretation of theentire text block.

In order to compare OCR results with the data included into the Postaldb, the Fuzzy Matching mechanism is used. For example, if OCR reads “SanDiego” as “San Dicgo” (‘c’ and ‘e’ are often misrecognized), FuzzyMatching will produce matching confidence above 80% between the two,which is sufficient to achieve the correct interpretation of OCR result.

2.5. Using Postal Database to Correct Addresses

After the interpretation of the address block was achieved, theindividual components will be corrected to become identical to thoseincluded into the Postal db. Optionally, the discrepancies betweenaddress printed on the bill and its closest match in Postal db could becorrected by replacing invalid, obsolete or incomplete data as follows:

Correcting ZIP+4

For example, 92128-1284 could be replaced by 92128-1234 if the latter isa valid ZIP+4 additionally confirmed by either the street address orpostal barcode, see 2.8

Adding Missing ZIP+4

For example, 92128 could be replaced by 92128-1234 if the latter is avalid ZIP+4 additionally confirmed by either the street address orpostal barcode, see 2.8

Correcting invalid street suffixes, such as “Road” into “Street” if the“Street” suffix can be confirmed by Postal db while the “Road” onecannot.

2.6 Computation of the Address Confidence

The system will assign a confidence value on the scale from 0 to 1000 toeach address it finds. Such confidences could be assigned overall forthe entire address block or individually to each address component(Recipient Name, Street Number, Apartment Number, Street Name, POBOXNumber, City, State and Zip). The larger values indicate that the systemis quite sure that if found, read and interpreted the address correctly.The component-specific confidence reflects the number of corrections inthis component required by process 2.5. For example, if 1 out of 8non-space characters was corrected the “CityName” address component(e.g. San Dicgo” v. “San Diego”), the confidence of 875 may be assigned(1000*⅞). The overall confidence is a weighted linear combination ofindividual component-specific confidences, where the weights areestablished experimentally.

2.7 Identification of Payee Vs. Payor

After one or more of address blocks have been captured as described in2.1-2.6, the system must make a determination as to which one is Payee'sand/or Payor's. The following factors help in such determination:

Presence of POBOX (it's much more likely to be a Payee than Payor ifPOBOX is present)

Location within the document (e.g. Payee is somewhat more likely to beprinted at the bottom, especially in the right/bottom corner)

Inclusion of certain words in the Recipient name item (some words like“Corporation”, “Department”, “Center” etc. indicate Payee)

Inclusion of frequent names in the Recipient name item (e.g. “John” ismore likely indicate Payor than Payee)

Adjacency to Postal barcodes, see 2.8. If one and only one of two foundaddresses is adjacent to a postal barcode, it is likelier to be Payor's.

Optional Payor hint, as explained in 2.9

Also, in a case when 3 or more addresses were found (and therefore morethan one address block compete for either Payee or Payor) the addressblock adjacent to postal barcode is given a preference.

2.8 Using Postal Barcode Reader

Postal barcodes are often printed on bills and they help to improveaccuracy of address capture.

The system uses Postal barcodes for 4 purposes:

To help in Payee vs. Payor identification (see 2.7)

To help choose the correct Payee or Payor when two found address blockscompete for the same field result

To correct ZIP-codes or capture them if they cannot be read from theimage due to poor quality. For example (see FIG. 6), if the address is603 illegible but the barcode 604 containing ZIP+4=“60094-4014” wasfound, the system can recreate City, State and ZIP fields from thebarcode.

To better detect address candidates, see 2.2

2.9 Using Payor Hint

Payor hint contain information about Payor (i.e. the bill's recipient).The system can use such information as one of the factors in Payee vs.Payor identification (see 2.7)

2.10 Using “Payment History” Hint

Such hint contains information about previously paid bills in theaccount. The system can use such information to significantly increaseaccuracy of capturing critical fields. Depending on which and how manycritical field values were included into the hint, the field captureerror may be reduced by 20-98% for repeating billers.

As an illustration, consider FIG. 6. Assume that the Bill Pay systemmade one recognition error (out of 16 digits) on the Account Numberfield 601 and recognized it as “4388 6755 5555 5551” as well as twoerrors (out of 17 characters) in the Biller's Name 602 and recognized itas “Chasc Card Servlces”. Let us assume that the system made no errorsin capturing the Payee ZIP-code 603 (out of 9 digits). In thisparticular example, even though the field itself cannot be read due topoor quality, the Postal Barcode 604 was captured correctly andpopulated the City, State and ZIP fields of Payee block.

If the “payment history” for this transaction included correct readingof Account Number (“4388 5755 5555 5551”), correct Biller's Name (“ChaseCard Services”) and correct biller's ZIP code (“60094-4014”), a standardfuzzy matching procedure will identify that 39 of 42 characters in all 3critical fields combined are matched correctly, resulting in about 92%matching confidence. If the system uses a threshold of 90% for thismatching (which could be made configurable), the errors in AccountNumber and Biller's Name may be corrected automatically.

2.11 Using Biller's Database

Available Biller's databases contain information about thousands ofbillers, including their postal addresses, names and account numberformats. This information allows to cross-validate payee information andother critical fields on CCBs, see [1].

III. Section 3 Capturing Balance Due from Credit Cards Bills

3.1 Keyword-Based Search

Balance Due field has a unique set of keyword phrases which allow toidentify the field's location on about 90% of CCBs. In remaining 10%,the keyword cannot be found due to some combination of poor imagequality, usage of small font, inverted text etc.

The most frequent keyword phrases are:

-   -   Balance Due    -   New Balance    -   New Balance Total    -   Outstanding Balance    -   Balance    -   Total Balance    -   Previous balance    -   Current Balance    -   Balance At Billing

These and other keywords could be printed in a single text line or twoadjacent lines (except for single-word ones)

Example on FIG. 7 shows a VISA Mileage Plus credit card bill with the“New Balance” keyword 701

Keywords are searched for in the OCR result using Fuzzy Matchingtechnique. For example, if OCR result contains “Bajance Due” then the“Balance Due” keyword will be found with confidence of 900 (out of 1000max) because 9 out of 10 non-space characters are the same as in the“Balance Due”.

3.2 Format-Based Search

Balance Due field has so-called “DollarAmount” format, which is one ofpre-defined data formats explained in [1]. Data format is used by BillPay system in combination with Keyword-based search 3.1 to furthernarrow down the set of candidates for the field.

Example on FIG. 7 shows a VISA Mileage Plus credit card bill with theBalanceDue data 702 adjacent to keyword 701. You can also see otherinstances of data with “DollarAmount” format in 703.

Each location of data found in proximity to keywords found in 3.1 isassigned the format-based confidence, which reflects how close data inthe found location matches expected format (in this case,“DollarAmount”).

3.3 Cross-Validation Against Codeline

On CCBs, Balance Due is always included into the Codeline. This allowsto use cross-validation technique similar to one explained in 1.2 forAccount Number field.

Example on FIG. 7 shows a VISA Mileage Plus credit card bill with theBalanceDue data 702 cross-validated using Codeline substring 704

3.4 Usage of the Largest Amount

If regular keyword-based search (see 3.1) doesn't yield results, thesystem can use the largest of all amounts included into the bill andfound by 3.2 as long as it can be validated against the codeline

Example on FIG. 7 shows a VISA Mileage Plus credit card bill with theBalanceDue data 702 is the largest of 3 “DollarAmount” fields (702 and703). It can also be cross-validated using Codeline substring 704

3.6. Confidence Score

Each result found by 3.1-3.4 is assigned a confidence score whichreflects how confident the system is that it found correct field result.A weighted linear combination of the following factors is used

-   -   the keyword finding confidence, see 3.1    -   the format-based confidence, see 3.2    -   the score reflecting geometrical alignment between keyword found        in 3.1 and data found by 3.2. For example, if the data is        located immediately below the keyword (with no characters in        between) like 701 and 702 on FIG. 7, or the data is located        immediately to the right from the keyword (again with no        characters in between) the score reaches its maximum value        of 1000. Various deviation from such alignment or presence of        characters in between cause lower scores.    -   if field's data could be found in the codeline, the confidence        gets an additional boost depending on how well the data and        codeline match.    -   the confidence is additionally boosted for larger amounts and        penalized for smaller ones.        The weight of each individual factor in the overall field        confidence score is established experimentally.        IV. Section 4 Capturing Biller's Name from Credit Cards Bills

4.1 Using Keywords

The Biller's name is often indicated on a bill by certain keywordphrases, the most frequent of which are:

-   -   Make check payable to    -   Make payment to    -   Made payable to    -   Remit Payments to    -   Check made payable to    -   Check or money order payable to

On FIG. 8, Biller's Name is pointed to by the keyword phrase “Make CheckPayable To” (801)

4.2 Finding Field Adjacent to Keyword

Once one or more such keywords were found, the system will try to findactual biller's name in the proximity of the keyword using the followingsequence of attempts:

1. Text immediately to the right of found keyword(s). Stop if text isfound, otherwise proceed to #2

2. Text immediately below found keyword(s). Stop if text is found,otherwise proceed to #3

3. Check if the Payee block is located below the keyword. If yes, takeits topmost line.

On FIG. 8, the Biller's Name “Citi Cards” 802 is located immediately tothe right of keyword 801

4.3 Cross-Correlation Against the “Recipient” Field in the Payee Address

On a large portion of CCBs the biller name is also included into the“recipient” field (the upmost line of the address block) of the Payeeaddress block. Therefore, the system will use the Payee's “recipient”field to cross-correlate (via Fuzzy Matching) with various biller's namealternatives found in 4.2 to choose the best candidate.

On FIG. 8, the Biller's Name “Citi Cards” 802 closely correlates with(or is identical to, if no OCR errors were made) Payee “recipient” field803

4.4. Using “Stop Words” to Limit the Field

When the Biller's name is found according to 4.1-4.2, sometimes anunrelated text is being added to it because it is printed to the rightfrom actual Biller's name. To identify and remove unrelated text, thesystem uses a set of so-called “stop-words”, which are commonly used togive the Payor some additional instruction related to paying the bill.

The list of commonly used “stop-words” include (but not limited to) thefollowing phrases

-   -   remit to    -   please do not    -   please return    -   and return    -   and indicate    -   do not send cash    -   in US funds    -   and include    -   and mail with    -   and mail to    -   please write    -   please make    -   please include    -   and write your    -   thank you    -   in US dollars    -   and this payment    -   in the enclosed    -   and write account    -   detach and exclude    -   payment due date

Consider the following example of using “stop-words”. On FIG. 8, theBiller's Name found according to 4.1-4.2 on the right from found keyword804 is 805 “Beaumont Hospital—DO NOT SEND CASH”. Then analysis finds a“stop-word” 806 (“DO NOT SEND CASH”), truncates the field result 805before the “stop-word”, hence producing the correct result 806(“Beaumont Hospital”)

4.5. Confidence Score

Each result found by 4.1-4.4 is assigned a confidence score whichreflects how confident the system is that it found correct field result.A weighted linear combination of the following factors is used

-   -   the keyword finding confidence, see 4.1    -   the geometrical relationship between keyword found in 4.1 and        data found by 4.2    -   cross-correlation to the Payee “recipient” field, established by        4.3

The weight of each individual factor is established experimentally.

4.6 Using Biller's Database

All candidates for biller's name captured from the bill according to4.1-4.4, get cross-correlated against all biller's names located at thecaptured biller's Zip-code (see Section 2). If one of the entries inBiller's db produces high match confidence against one of results4.1-4.4, the latter will be chosen as the correct biller's name, see[1]. The matching threshold is configurable. If none of Biller db entrymatches to field results found by 4.1-4.3, the result with the highestconfidence score 4.5 will be used.

Section 5 Capturing AccountNumber on Major Credit Cards

There is a set of rules applicable to all Major Credit Cards (MCC forshort) which help to increase recognition accuracy on such bills.

5.1 Limitations of AccountNumber's Leading Digits

MCCs (Visa, MasterCard, AmEx, Diners, and Discover) have well-definedaccount number formats. Their account numbers start with a particulardigit or a narrow range of digits (say, Visa always starts with ‘4’,MasterCard with “51”-“55” and so on). This limitation translates innarrowing the AccountNumber's format, see 1.2

5.2 Limitations of AccountNumber's Length

AccountNumber's length is also restricted. The length depends on thecredit card, 16 digits length is the most often case. This limitationalso translates in narrowing the AccountNumber's format, see 1.2

5.3 Mod 10 Rule (LUHN Formula)

Account Number field on MCCs satisfied LUHN Formula (Mod 10) rule, whichwe included below for reference.

The following steps are required to validate the account number on MCCs:

Step 1: Double the value of alternate digits of the account numberbeginning with the second digit from the right (the first right—handdigit is the check digit.)

Step 2: Add the individual digits comprising the products obtained inStep 1 to each of the unaffected digits in the original number.

Step 3: The total obtained in Step 2 must be a number ending in zero(30, 40, 50, etc.) for the account number to be validated.

5.4 Detection of Account Number Entirely by Codeline

If the fact that bill is issued by a major credit card was established,the system can in most cases find the field directly in Codeline w/o anecessity to do the OCR. This becomes possible if a single substring inthe codeline satisfies all restrictions 5.1-5.3

IV. Capturing Biller's Address from Credit Cards Bills

Using Text Blocks to Find Possible Addresses

Usually, addresses are printed as left-, right- or center-justified textblocks isolated from the rest of document text by significant whitemargins. Based on this information, the system may detect potentialaddress locations on a document by building text block structure.

Filtering the Text Block by the City/State/ZIP Line

In most of US addresses, the bottommost line contains City/State/ZIPinformation. The system can utilize this knowledge by filtering out thetext blocks found in 2.1 which do not have enough alphas (to representCity and State), do not contain any valid state (which is usuallyabbreviate to 2 characters) and/or do not contain enough numbers in theend to represent Zip-code.

Using Postal Database and Fuzzy Matching to Interpret Addresses

Once address candidates are selected using 2.1 and 2.2, the system canbuild the entire address block starting with City/State/ZIP at thebottom line and including 1-3 upper lines as potential Name and StreetAddress components. Since the exact format of the address is notwell-defined (it may have 1-4 lines, be with and without names, be withand without POBOX etc), the system has to make multiple addressinterpretation attempts to achieve satisfactory interpretation of theentire text block.

In order to compare OCR results with the data included into the Postaldb, the Fuzzy Matching mechanism is used. For example, if OCR reads “SanDiego” as “San Dicgo” (‘c’ and ‘e’ are often misrecognized), FuzzyMatching will produce matching confidence above 80% between the two,which is sufficient to achieve the correct interpretation of OCR result.

Using Postal Database to Correct Addresses

After the interpretation of the address block was achieved, theindividual components will be corrected to become identical to thoseincluded into the Postal database.

Computation of the Address Confidence

The system will assign a confidence value on the scale from 0 to 1000 toeach address found above. Such confidences could be assigned overall forthe entire address block or individually to each address component(recipient name, street number, apartment number, street name, POBOXnumber, City, State and ZIP). The larger values indicate that the systemis quite sure that if found, read and interpreted the address correctly.

Identification of Payee Vs. Payor

After one or more of address blocks have been captured, the system mustmake a determination of which one is Payee and Payor. The followingfactors help in such determination:

Presence of POBOX (it is much more likely to be a Payee than Payor ifPOBOX is printed).

Location within the document (e.g. Payee is somewhat more likely to beprinted at the bottom, especially in the right/bottom corner)

Inclusion of certain words in the Recipient name item (some words like“Corporation” indicate Payee)

Inclusion of frequent names in the Recipient name item (e.g. “John” ismore likely indicate Payor than Payee)

Adjacency to Postal to barcodes (if more than 1 block competes foreither Payee or Payor, the one adjacent to a barcode wins).

Using Postal Barcode Reader

Postal barcodes are often printed on bills and they help to improveaccuracy of address capture.

The system uses Postal barcodes for 3 purposes:

To help in Payee vs. Payor identification (see 2.6)

To correct ZIP-code

To better detect address candidates

Using Payor Hint

Payor hint contain information about Payor (i.e. the bill recipient).The system can use such information for Payee vs. Payor identification(see also 2.6)

Using Payee Hint

Payee hint contain information about existing billers in the account.The system can use such information to significantly increase accuracyof capturing critical fields. Depending on which and how many criticalfield values were included into the hint, the field capture error may bereduced by 20-98% for the pre-existing (i.e. repeating) billers.

Using Biller's Database

Available Biller's databases contain information about thousands ofbillers, including their postal addresses, names and account numberformats. This information allows to cross-validate payee information andother critical fields on CCBs.

III. Capturing Balance Due from Credit Cards Bills

Keyword-Based Search

Balance Due field has a unique set of keywords which allow us toidentify the field's location on about 90% of CCBs. In remaining 10% thekeyword cannot be found due to some combination of poor image quality,usage of small font, inverted text etc.

Cross Validation Against Codeline

On CCBs, Balance Due is always included into the Codeline. This allowsus to use a cross-validation technique by comparing content from twodifferent fields that should be identical.

Usage of the Largest Amount

If regular keyword-based search (see 3.1) does not yield results, thesystem can use the largest of all amounts included into the bill as longas it can be validated against the codeline.

IV. Capturing Biller Name from Credit Cards Bills

Using the “Recipient” Field in the Payee Address

On a large portion of CCBs the biller name is included into the“recipient” field in the Payee Address. Therefore the system will usethe “recipient” field as a candidate for the biller's name

Using Keywords

The Biller's name is often indicated on a bill by certain keywords, like“Pay to”, “Make your check payable to” etc. Once one or more suchkeywords were found, the system will try to find actual biller's name inthe proximity of the keyword using the following sequence of attempts:

-   -   1. Text immediately to the right of found keyword(s). Stop if        text is found, otherwise proceed to #2.    -   2. Text immediately below found keyword(s). Stop if text is        found, otherwise proceed to #3.    -   3. Check if the Payee block is located below the keyword. If        yes, take its topmost line.

The system will use the text found in 1-3 above as another candidate forthe biller's name in addition to the one presented in the sectionimmediately above.

Using Biller's Database

All candidates for biller's name get cross-correlated against allpossible billers located at the biller's Zip code found (see Section 2).The entry in Biller's db with the highest match confidence will bechosen as the correct biller.

V. Capturing Account Number on Major Credit Cards

There is a set of rules applicable to all Major Credit Cards (MCC forshort) which help to increase recognition accuracy on such bills.

Limitations on Character Set

Account Number field in all MCCs is purely numeric (unlike say Insurancebills which may include alphas).

Limitations of Account Number's Leading Digits

MCCs (Visa, MasterCard, AmEx, Diners, and Discover) have well definedaccount number formats. Their account numbers start with a particulardigit or a narrow range of digits (say, Visa always starts with ‘4’,MasterCard with “51”-“55” and so on).

Limitations of Account Number's Length

Account number length is also well restricted. The length depends on thecredit card, 16 digits length is the most often case.

Mod 10 Rule (LUHN Formula)

Account Number field on MCCs satisfied LUHN Formula (Mod 10) rule, whichwe included below for reference.

The following steps are required to validate the account number:

Step 1: Double the value of alternate digits of the account numberbeginning with the second digit from the right (the first right—handdigit is the check digit.)

Step 2: Add the individual digits comprising the products obtained inStep 1 to each of the unaffected digits in the original number.

Step 3: The total obtained in Step 2 must be a number ending in zero(30, 40, 50, etc.) for the account number to be validated.

Detection of Account Number Entirely by Codeline

If the fact that bill is issued by a major credit card was established,the system can in most cases find the field directly in Codeline w/o anecessity to do the OCR.

VI. Overall Flowchart of Capturing Critical Fields from Credit CardBills

Description of Overall Flowchart

FIG. 8 is a flowchart of a method of capturing critical fields in acredit card bill, in accordance with embodiments of the invention. Inone embodiment, the steps for capturing critical fields include:

100—Input binary image. It is created as a result of processing themobile color JPG-image of the bill as described in Patent [1].

150—Codeline recognition.

200—ASCII string representing result of 150

250—Postal Barcode recognition

300—Postal Barcodes

350—Applying Dynamic Capture module [1] to find alternatives for AccountNumber and Balance Due based on the field's keywords (such as “AccountNumber”, “Balance Due” etc) and field's format

400—Final Account Number and Balance Due results based oncross-validation against codeline 200

450—Postal database

450—Address recognition and validation using Postal db 450

500—Payee and Payor address block as a result of 450

550—Identification of Payee and Payor

600—Payee block

650—Finding candidates for Biller Name using keywords and payee

700—Biller name candidates

750—Biller database

800—Validation candidates 700 against database 750

850—Final Biller name

FIG. 9 is one embodiment of a network and system upon which the methodsdescribed herein may be implemented, including a capture device 702which captures an image 704 of a credit card bill, then transmits itover a network 706 to a server 708 for processing. In one embodiment,the capture device 702 also performs one or more of the processing stepsdescribed herein in addition to, or instead of, the server 708.

FIG. 10 is an embodiment of a computer, processor and memory upon whicha mobile device, server or other computing device may be implemented tocarry out the methods described herein. The server 708 may include apower supply 902, processor 904, network interface module 906, memory908 and a CCB recognition module 910 for performing the specific creditcard bill recognition and identification steps described herein.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notof limitation. The breadth and scope should not be limited by any of theabove-described exemplary embodiments. Where this document refers totechnologies that would be apparent or known to one of ordinary skill inthe art, such technologies encompass those apparent or known to theskilled artisan now or at any time in the future. In addition, thedescribed embodiments are not restricted to the illustrated examplearchitectures or configurations, but the desired features can beimplemented using a variety of alternative architectures andconfigurations. As will become apparent to one of ordinary skill in theart after reading this document, the illustrated embodiments and theirvarious alternatives can be implemented without confinement to theillustrated example. One of ordinary skill in the art would alsounderstand how alternative functional, logical or physical partitioningand configurations could be utilized to implement the desired featuresof the described embodiments.

Furthermore, although items, elements or components may be described orclaimed in the singular, the plural is contemplated to be within thescope thereof unless limitation to the singular is explicitly stated.The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent.

What is claimed is:
 1. A computer readable medium containinginstructions which, when executed by a computer, perform a processcomprising: receiving an input image of a credit card bill; performing acodeline recognition process; capturing a postal service barcode;extracting an account number and balance due from the image using anoptical character recognition process; obtaining an address of a payorand payee from the image; identifying the payor and payee based on acomparison of the obtained addresses with a database of payor and payeeaddresses; performing a keyword-based capture to obtain a biller name;confirming a biller name; and outputting bill data from the credit cardbill to a user.